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Introduction 


Get the Picture?! Guiding and Engaging Exceptional Teens (Get the Picture?!) was a four year 
Investing in Innovation (i3) Development grant. The project, as implemented by the Green River Regional 
Educational Cooperative (GRREC), was a multi-layered initiative that focused on building capacity in both 
adults and students within the schools being served. For the two years prior to project implementation, 
GRREC special educators worked alongside the teachers, school and district leaders, and specialists 
from the Kentucky Department of Education (KDE) to move Franklin-Simpson High School in rural south 
central Kentucky from Persistently Lowest Achieving status to one of two designated Hub Schools in the 
state. Get the Picture?! was designed to replicate that work in nine participating schools in nine rural, 
high-poverty Kentucky school districts. The purpose of this study was for the Center for Research in 
Educational Policy (CREP) to analyze student achievement and behavior data to evaluate the 
effectiveness of Get the Picture?! in developing the self-determination skills of participating students with 
special needs after four years of implementation. 


CREP is a State of Tennessee Center of Excellence, located at the University of Memphis, whose 
mission is to implement a research agenda associated with educational policies and practices in K-12 
schools and higher education (HE) as well as community outreach programs, and to provide a knowledge 
base for use by educational practitioners and policymakers. Since 1989, CREP has served as a 
mechanism for mobilizing community and university resources to address educational problems and to 
meet the University's commitment to primary and secondary schools. Functioning as a part of the 
College of Education, CREP seeks to accomplish its mission through a series of investigations conducted 
by Center personnel, college and university faculty, and graduate students. 


Background 

As shown in Figure 1 below, the logic model for Get the Picture?! had three sets of outcomes 
(Short, Medium, and Long) that were theorized to result as a function of the inputs and activities outlined. 
The current report focuses on the student outcomes, and in particular, improved academic achievement 
and behavior under the Medium and Long-Term outcomes. 


Program Description 

Through the development of self-determination skills, the goal of the intervention was to increase 
the number of students with disabilities who achieved the state standard for College and/or Career 
Readiness by meeting established benchmarks on State/National assessments (ACT, COMPASS, 
KYOTE, KOSSA, WorkKeys, ASVAB) and/or completion of a recognized industry certification (Microsoft, 
Welding, CNA, AutoCAD, etc.) in each of the nine participating schools. However, it should be noted that 
the state of Kentucky changed its student accountability model during the grant, which impacted the 
state/national assessments administered. As a result, all of the originally established benchmarks were 
not available at the conclusion of the grant at the end of 2019. 
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Figure 1. Get the Picture?! Logic Model 
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Research Questions 

The main goal of Get the Picture?! was to increase the college/career readiness of exceptional 
students. The primary outcomes initially examined included the Kentucky Department of Education’s 
(KDE) approved indicators of readiness, which consisted of standardized assessments and industry 
certificates. Additional outcomes originally intended to be examined in the study beyond the KDE 
indicators included self-management behaviors, dual-credits, and graduation. The original six research 
questions and outcome domains for the study were as follows: 


1. Does Get the Picture?! have an effect on the College Readiness of 12th grade students with 
disabilities compared to the business-as-usual condition [College Readiness Domain]? 

2. Does Get the Picture?! have an effect on Career Readiness of 12th grade students with 
disabilities compared to the business-as-usual condition [Career Readiness Domain]? 

3. Does Get the Picture?! have an effect on the College and Career Readiness of 12th grade 
students with disabilities compared to the business-as-usual condition [College and Career 
Readiness Domain]? 

4. Does Get the Picture?! have an effect on self-management behavior of students with disabilities 
throughout high school when compared to the business-as-usual condition [Self-Management 
Behaviors Domain]? 

5. Does Get the Picture?! have an effect on dual-credit courses for which credit was earned by 
students with disabilities throughout high school compared to the business-as-usual condition 
[Dual Credits Domain]? 

6. Does Get the Picture?! have an effect on high school students with disabilities graduating with a 
regular diploma in four years or less compared to the business-as-usual condition [Graduation 
Domain]? 
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However, due to changes in the Transition Readiness Standards under the Kentucky 
Accountability System that took place in the 2018-19 school year, the evaluation team had to revise the 
outcomes and outcome domains to align with the revised standards, which are shown in Appendix A. 
Instead of the original six outcomes and domains, only contrasts in two revised domains were of interest 
for the final impact analyses: Transition Readiness and Self-Management Behaviors. Transition 
Readiness reflects a name and compositional change from the original three College and Career 
contrasts to a single contrast with two different pathways to success (Academic or Career Readiness). 
Therefore, having Transition Readiness as one contrast would follow the state’s new approach. In 
addition, the state no longer had a combined category (i.e., Academic and Career Readiness), and thus 
the evaluation team dropped the contrast for College and Career Readiness. 


In addition, the tests/measures that comprised the Transition Readiness outcome under the 
2018-19 standards were different from those under the original College Readiness and Career Readiness 
contrasts. Notably, dual-credit courses and graduation became part of the Transition Readiness indicator 
and therefore were dropped as separate contrasts. Finally, the main contrast for Self-management 
Behaviors was now comprised of the cumulative number of in-school suspensions after four years. 
Table 1 below shows the final Contrast/Outcome Domains table incorporating the revisions. 
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Table 1. Revised Contrast/Outcome Domains 
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Intervention and Comparison Condition 


Description of the Intervention Activities 

Each student with disabilities in the treatment schools received one-on-one support through a 
school-based Career Strategist, meeting at least weekly to monitor the student’s Individual Learning 
Plan, which focused on college and/or career readiness with realistic, big picture goals and action steps 
for life after high school. Together, the student and Career Strategist ensured the student’s aspirations 
and aptitudes informed his/her personalized career pathways, and that coursework supported and aligned 
to their career plan. The Strategist also worked with the student to address barriers to success. 

Each treatment school also had a school-based Support Team —the leaders (College/Career 
Readiness (CCR) Coaches, Lead Career Strategist, and Career Strategist), special education teachers, 
and counselors — that coordinated and monitored one-on-one work and the Student-to-Strategist match. 
Each local Support Team, guided by the CCR Coach, met at least four times per academic year to make 
data-driven decisions to support college and career readiness for all students with disabilities. The 
school-based Support Team worked alongside the CCR coach to coordinate the project at the school 
level. Each team developed a manageable plan that allowed each school to utilize the Get the Picture?! 
components to best serve individual student needs. CCR Coaches provided CCR focused instruction to 
students while supporting the Lead Strategist, the Career Strategist and other educators. In addition, 
CCR Coaches worked directly in the participating schools providing ongoing, embedded support and 
modeling, including student strategies, family facilitation, and data based decision-making. The key 
activities and inputs delivered to participants as part of the intervention are listed in Table 2 below. 


Table 2: Key Activities/Inputs Delivered to Get the Picture?! Participants 


ey Description 
1 Needs Assessment 
2 Culture Survey 
3 Program Supports for the College/Career Readiness (CCR) Coaches 
4 Program Supports for Teachers 
4b Teacher Participation 
5a Program Supports for Strategists 
Sb Strategist Participation/Activities 
6a Program Support for School-based Support Teams 
6b Participation/Activities of School-based Support Teams 
7a Community and Family Engagement Opportunities Provided by Schools 
7b Community and Family Participation 


Participants in the Intervention 


Eligibility Criteria 

All students (a) identified as receiving special education services in the nine treatment and 18 
control schools who (b) began 9th grade in Year 1 of implementation (2015-16) and (c) were working 
toward a regular high school diploma were eligible to be included in the confirmatory impact study. Any 
student in the nine treatment schools who met the eligibility criteria was invited to participate and offered 
an opportunity to voluntarily enroll to participate in Get the Picture?! Comparison group members in 
control schools received business-as-usual support with the understanding that all Kentucky schools are 
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focused on college and career readiness to some extent. Comparison/Control schools could not receive 
support around any of the Key Components. 


Setting Location and Characteristics 

The nine participating high schools served by Get the Picture?! were chosen based on four 
selection criteria: 

1. Treatment schools had to be in one of the 37 school districts located in 26 counties across South 
Central Kentucky included in the Green River Regional Educational Cooperative (GRREC) and 
be among the lowest performing in terms of College and Career Readiness (CCR). 

2. GRREC eliminated from consideration districts that were currently involved in major ongoing 
initiatives that might interfere with project outcomes. 

3. Data from students with disabilities in each of the potential schools was reviewed to identify 
schools with large gaps between students with disabilities and other students in terms of 
graduation rates and other college and career readiness indicators. 

4. GRREC invited 12 school districts and their high schools to participate. Of that number, the nine 
schools that were most eager to participate were selected as the treatment schools. 

Get the Picture?! was a four year i3 Development grant, with interventions implemented in the 
nine treatment schools in the 2015-16 through 2018-19 academic school years. As the grant ended in 
December 2019, a no-cost extension was requested and granted to allow time for the confirmatory 9th 
grade cohort of students to complete up to four years of treatment and for outcome data for the 2018-19 
school year to become available and analyzed. The last outcome data files were received from the 
Kentucky Department of Education in March 2020. 


Study Design and Measures 


Method of Assignment of Units to Conditions 

As noted in the Setting Location and Characteristics section above, clusters (i.e., schools) were 
assigned to treatment or comparison groups non-randomly, and blocking was not used as part of the 
assignment process. The initial control school pool was comprised of all public high schools in the state 
of Kentucky containing 9th through 12th grades who served students with disabilities and were not 
receiving the treatment. To remain in the control school pool, the following three selection criteria were 
applied. First, schools had to be traditional high schools. This meant the following types of schools were 
removed: 

e =6Alternative schools 

e Detention centers 

e Development centers 

e Treatment centers 

e = Technical high schools 

e Other non-traditional high schools. These schools were missing large portions of the data for 
matching and had enrollment below 100 students. 

Second, schools that (a) had similar activities to Get the Picture?! (GTP) in place as part of 
another GRREC project or (b) that had received similar support from GRREC at the request of the district 
were not eligible to be in the pool of control schools. Finally, schools in the control pool had to have 
demographic and achievement data available on the Kentucky School Report Card for matching 
purposes. 

Once the final control school pool was determined, each of the nine participating GRREC 
treatment schools was matched to two control schools using nearest neighbor propensity scores based 
on school level data from the 2013-14 Kentucky School Report Card (the most recent year available). 
School matches were finalized in September 2015 after one of the initial control schools selected was 
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replaced after being identified by GRREC as recently receiving specific support from GRREC on college 
and career readiness and generational poverty. The factors used to match the schools were: 

e Whether the school was rural 

e Location (latitude and longitude) 

e =Total enrollment 

e Average attendance 

e Ethnic breakdown (percentage non-white) 

e Gender breakdown (percentage female) 

e Percentage of students on free/reduced lunch 

e Percentage of students with disabilities 

e Graduation rate (overall for the school) 

e College and Career Readiness rate 

e Composite ACT score average, and 

e Composite high school K-PREP scores (i.e., average proficiency rate across all available state 

assessment subject areas). 

The nine treatment schools were compared to 214 control schools, and the two best matches for 
each treatment school were determined based on having the most similar propensity score. As shown in 
Table 3, the final number of schools included in the study sample was 27 (9 treatment and 18 matched 
controls). Students identified as receiving special education services in the nine GTP treatment and 18 
control schools who began 9th grade in Year 1 of implementation (2015-16) and who were working 
towards a regular high school diploma were included in the study for the confirmatory analysis (i.e., main 
impact analysis). To be included in the treatment sample, students had to attend a treatment school and 
be served by GRREC in 2015-16, no matter if they were (a) served in any later years, (b) changed grades 
unexpectedly (e.g., move from 9"" to 11" grade), or (c) graduated in less than four years. 

Students who transferred between the same type of school (i.e., either treatment or control) were 
included in the analyses. Any student who transferred between different types of schools (i.e., between a 
treatment and control school and vice versa) were excluded, as well as any students who switched back 
and forth between types of schools (e.g., treatment, control, treatment). This included a total of 3 GTP 
and 41 control students. Special education students in the comparison schools received the “business- 
as-usual” supportive services; noting, however, that all public schools in Kentucky were focused on 
College and Career Readiness to some extent. 
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Table 3. Treatment and Control School Matches Based on 2013-14 School Year Report Card Data 


SYe1aTeye) 


Treatment 
School 1 
Control 


Control 
School 1b 
Treatment 
School 2 


School 2a 
Control 
School 2b 
Treatment 
School 3 
Control 
School 3a 
Control 
School 3b 
Treatment 
School 4 
Control 


School4a 0 


Control 
School 4b 
Treatment 
School 5 
Control 


School sa 


Control 
School 5b 


Percent 
Non- 
Minority 


Percent 
Female 


rNuc=varer=lare) 


Enrollment pate 


95.2 52.3 


1,814 


803 93.4 96.1 


93.0 


1,071 


98.4 49.0 


mas) Rexcy al 
Free/Reduced 
UT area) 


55.6 


Percent 
Special 
/=xo [Uler= cela] 


Graduation 
atelis) 


Percent 
College/Career 
Ready 


ACT K-PREP 
Composite Composite 


19.8 


Note. Each treatment school is listed first, shaded and in bold text, with the control schools ordered based on the closeness of the propensity 
score match to the respective treatment school. Pseudonyms are used to protect the identity of individual schools. 
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Table 3 (Continued) 


Percent Percent Percent 


Free/Reduced _ Special ions na College/Career 


ACT K-PREP 
Composite Composite 
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Treatment 
School 6 1,465 
Control 
School 6a 
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School 6b 
Treatment 
School 7 1,031 
Control 
School 7a 1,290 94.7 98.1 50.0 45.87.9 
Control 
School 7b 1,019 59.7 50.6 82.1 
Treatment 
School 8 
Control : 
School 8a 210 94.1 94.8 52.4 
Control 
School 8b 1,050 63.2 50.6 
Treatment 
School 9 
Control ' 
School 9a 852 89.0 96.8 47.4 
Control 
School 9b 1,250 94.0 95.6 49.2 

Note. Each treatment school is listed first, shaded and in bold text, with the control schools ordered based on the closeness of the propensity 

score match to the respective treatment school. Pseudonyms are used to protect the identity of individual schools. 


8.5 
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Baseline and Outcome Confirmatory Measures 

The Kentucky Department of Education (KDE) provided CREP with student-level data files that 
contained a student ID number, 8th grade student demographics, and the associated baseline and 
outcome data for each special education student in the treatment and control schools. Additionally, Get 
the Picture?! (GTP) program personnel provided CREP with a list of student numbers representing 
students who received GTP services during the project. CREP used the GTP student lists to ensure we 
received the correct treatment student outcome data from KDE. As a result, three GTP students who 
were not in the KDE files were deleted. 

The initial sample size consisted of 246 possible GTP treatment and 832 possible control 
students. After deleting students with alternative diplomas and missing baseline or outcome data, there 
were a total of 93 GTP 9" grade treatment students and 345 9" grade control students available for 
analysis. For the confirmatory analyses, there were two outcome variables in two different outcome 
domains: 

e =Transition Ready, a binary “Yes”/"No’” variable [Transition Readiness domain], and 
e The cumulative number of in-school suspensions (a continuous variable) [Self-management 
behaviors domain] 

As there were two different outcomes in two different domains (one outcome per domain), no 
adjustments were made for multiple comparisons. As shown in Appendix A, a student was flagged by 
KDE as Transition Ready if they (a) earned a high school diploma by meeting/exceeding the Kentucky 
Minimum High School Graduation Requirements and (b) met the KDE requirements for either Academic 
or Career Readiness. It should be noted that the What Works Clearinghouse (WWC) considers these 
types of outcomes, which include standardized tests outcomes measured using administrative data to be 
reliable (What Works Clearinghouse, 2020). Baseline equivalence was determined using variables 
measured when students in the analytic sample were in 8th grade. The pretest variable for the 
cumulative in-school suspensions outcome was the total number of 8th grade in-school suspensions. 
As there was not a natural pretest for the Transition Ready outcome, the pre-intervention measure of 
academic performance or achievement was determined by selecting the ACT EXPLORE subtest 
(Rhetorical Skills) with the strongest correlation (point biserial) with the Transition Ready outcome (fpp = 
0.11, p = .027), which was used in combination with the pre-intervention measure of special education 
eligibility (IEP) as the Transition Ready pretest measures. 


Analytic Approach 


Confirmatory Outcomes 

For the confirmatory analyses, outcome data were examined using two-level Hierarchical Linear 
Models (HLM) (for Cumulative In-School Suspensions) and Hierarchical Generalized Linear Models 
(HGLM) (for Transition Ready) to account for the nested structure of the data (i.e., students nested within 
schools). In addition to the baseline pretest covariates (EXPLORE Rhetorical Skills and IEP or 8th Grade 
in-school suspensions), each model included the 8th grade demographic binary covariates for female 
status, minority status, and Free/Reduced Priced Lunch (FRL) status. To be included in the confirmatory 
analyses, GTP and control students had to have both baseline and outcome data in the KDE file (i.e., no 
missing data). 


Statistical Model for Estimating Baseline Equivalence for Confirmatory Outcomes 
The mixed model equation for the confirmatory baseline analysis is presented below, with results 


presented in Table 4. 


8" Grade EXPLORE and IEP or Suspensions; = yoo + Yor* Group; + uojt Fj 
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Statistical Model for Estimating Confirmatory Outcome Impacts 
The mixed model equations for the confirmatory outcome analyses are presented below, with 
results presented in Table 5. 


Transition Ready Outcome’ 


Ni = Yoo + yor* Group; 
+ v10* FEMALE 8thGradej 
+ y20* MINORITY 8thGradej 
+ y30* FRL8th Grade; 
+ y4o*EXPLORE Rhetorical Skills; 
+ Uo 


Cumulative Suspensions Outcome 


CumulativeSuspensionsj = yoo + yo1* Group; 
+ y1o* FEMALE 8thGradej 
+ y2o*MINORITY8thGradej 
+ y30* FRL8th Grade; 
+ y4o"lSS8thGrade; 
+ Ugt Fj 


Study Results 


Confirmatory Outcome Impacts 

As shown in Table 4, there were no statistically significant (p < 0.05) or nonequivalent effect size 
differences (i.e. g > 0.25) between 9'" grade cohort GTP treatment and control students at baseline for 
either the EXPLORE Rhetorical Skills subtest, the 8th grade total number of in-school suspensions, or the 
8" grade percent of special education students. However, both impact analyses adjusted for the baseline 
covariates (except IEP), particularly as the effect size for 8" grade suspensions was in the range for 
statistical adjustment to satisfy the baseline equivalence requirement (0.05 < |Baseline ES| < 0.25). 


Table 4. Confirmatory 9° Grade Cohort Outcomes Baseline Equivalence Assessment 


Comparison Group Treatment Group 
Treatment 
Sample Sample feXe} a} axe) Standardized 
Measure size Mean (12)) size difference oT aCcYc:Valer-m (e)) 
EXPLORE 
Rhetorical 345 5.35 (1.65) 93 5.28 (1.61) -0.07 0.01 
Skills 
8" Grade 
Number of 345 0.80 (2.42) 93 42 (1.09) -0.38 -0.19 
Suspensions 
8" Grade 
Percent 
Special 345 100% NA 93 100% NA 0 0.0 
Education 
*p<0.05 


1 |EP was not included as a covariate for the Transition Ready outcome since all treatment and control students 
were classified as receiving IEP services. 
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As shown in Table 5, GTP treatment students were statistically significantly more likely (p < 0.05) 
to be Transition Ready compared to control students. An odds ratio of 2.15 indicated that GTP students 
were more than twice as likely to achieve Transition Readiness status compared to control students. In 
addition, the effect size (CIES = 0.46) indicated that GTP students achieved Transition Readiness status 
at a rate better than 68% of control students. However, there was no statistically significant difference 
between GTP treatment and control students in the Cumulative Number of Suspensions. 


Table 5. Confirmatory 9" Grade Cohort Outcomes Impact Analysis Results 


Comparison Group Treatment Group 
"foe (=) Bs Treatment 

Outcome Sample Sample | adjusted — control Standardized 

measure size size bl=x= 10) difference | difference (g) 
Transition 0 0 * 
Ready 345 23% NA 93 39% NA 16% 0.46 0.03 
Cumulative 
Suspensions 345 4.26 (8.56) 93 3.33 (4.53) -0.93 -0.12 0.17 


* p < 0.05; Note: Analyses included 9 treatment and 17 control schools 
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Summary 


Overall, after four years of implementation, Get the Picture?! was able to demonstrate a 
statistically significant positive impact on the Transition Readiness of participating 9" grade 
cohort students compared to controls. Treatment students had statistically significantly higher odds of 
being Transition Ready, and were more than twice as likely to achieve Transition Readiness status 
compared to control students. However, while the confirmatory study showed the intervention was also 
able to reduce the total number of in-school suspensions for treatment students relative to controls, the 


outcome was indeterminate (i.e., not statistically significant). 
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Appendix A: Revised Transition Readiness Standards 
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Transition Readiness 


Student Expectations for Transition Readiness 


High School Diploma 


06/03/2019 


Earn a high school diploma by meeting/exceeding the Kentucky Minimum High School Graduation 


Requirements 


AND 


Meet Requirements of Academic or Career Readiness 


fo 


Academic Readiness 


¥ Benchmarks, determined by Council on 


Postsecondary Education (CPE) ona 
college admissions exam or college 
placement examination; 

OR 


¥  Agrade of C or higher in each course on 6 


hours of KDE-approved dual credit; 
OR 


Y Ascore of 3+ on exams in 2 Advanced 


Placement courses; 
OR 

¥ Ascore of 5+ on 2 exams for 
International Baccalaureate courses; 
OR 

¥ Benchmarks on 2 Cambridge 
Advanced International 
examinations; 
OR 


¥ Completing a combination of academic 


readiness indicators listed above. 


Career Readiness 


v Receiving an Industry 
Certification (Approved by the 
Kentucky Workforce Innovation 
Board on an annual basis); OR 

Y Scoring at or above the 
benchmark on the Career and 
Technical Education End-of- 
Program Assessment for 
articulated credit; 

OR 

¥ Agrade of C or higher in each 
course on 6 hours of KDE- 
approved Career and Technical 
Education dual credit; OR 

¥ Completing a KDE/Labor 
Cabinet- approved 
apprenticeship; 

OR 

¥ Completing a KDE-approved 
alternate process to verify 
exceptional work experience. 


e Demonstration of academic readiness shall 
include one quantitative reasoning or natural 


sciences and one written or oral 


communication, or visual and performing arts; 


or humanities; or social and behavioral 
sciences learning outcomes. 


| English Language 


Readiness (only 
required for 
English Learners) 


Y Meeting exit criteria 
for English language 


proficiency 


assessment (Overall 


composite of a 
4.5 on a Tier B/C) 
for any student 
who received 
English Language 
services during 
high school. 


e English Language 
Learners are included 


in academic and 


career readiness in 


addition to English 
Language 
Readiness. 


Note: Students participating in the alternate assessment program and earning an alternate diploma will have criteria 
for Transition Readiness based on alternate assessment requirements and employability skills attainment. 


Please contact the Office of Standards, Assessment and Accountability (OSAA) if there 


KDE:0SAA:06/3/2019 


are any questions: 
(502) 564-4394 
dacinfo@education.ky.gov 
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Get the Picture?! 


Fidelity of Implementation Final Report 


Roland O’Daniel, PhD. CTL 


Introduction 
The four-year implementation began in 2015. Work with students | 
began with the confirmatory cohort as freshmen and expanded to Sin 
include all four classes through the spring of 2019. For the purposes . 
of this Fidelity of Implementation report, data is at the project level for each year and not solely 
for the confirmatory cohort. The purpose of the implementation evaluation was to: 
o Establish efficacy of the Get the Picture?! Intervention model in ensuring that students 
with disabilities graduate from high school ready for college and career, 
o Meet WWC evidence standards, and 
o Provide ongoing feedback on fidelity of implementation, allowing GRREC to refine the 
intervention model in order to meet project goals. 


Fidelity of implementation visits to sites were completed with in-person visits in years 1 and 2 
and virtual visits in years 3 and 4. The same data was collected during all four implementation 
visits to each site. The visits contained focus groups of randomly selected students and 
teachers, interviews with administration, and randomly selected observations of student and 
strategist sessions. Site visits were conducted between February and May across all 
participating schools each year. Additional data were collected and shared through a secured 
network. Visits were conducted by the following CTL staff: Dr. Roland O’Daniel, Dr. Ashley 
Perkins, and Ms. Rita Messer. This report outlines Get the Picture’s progress toward reaching 
fidelity of implementation across the Key Components and activities as outlined in the Get the 
Picture logic model and fidelity of implementation matrix. 


The report will follow the outline provided by NEi3 Criteria and Procedures. The report begins 
with an analysis of each key component, the logic model developed during the beginning of the 
project, and is followed by the fidelity of implementation matrix which identifies each indicator 
within each component. 


Key Component Analysis 
Needs Assessment 
During the first nine weeks of each project year, all districts conducted a needs assessment to 
review strategist training and expectations, student enrollment and success plans, and 
monitoring interactions with students. Data retreats were conducted for all schools, which 
consisted of one day of data analysis. 
All schools had an Initial Implementation plan in place with updates made regarding yearly plans 


to meet project goals. By the end of January each year, all districts had conducted follow-up 
meetings and updated plans with progress indicators toward meeting the yearly goals. 
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Culture Survey 


Each year, surveys were conducted for students, parents, and faculty. Evidence in coaching 
logs, school implementation plans, and PD offerings indicate data from the surveys impacted 
professional learning and supports offered through the project during year two through four of 
implementation. 


In years one through three, there is an alignment between participating students across the 
different components of the grant. Evaluators have proxy IDs for students from Student Success 
Plans (SSP) and from Strategist logs (CCR). From the SSP and CCR lists, obvious typos were 
removed (too many or too few digits) and IDs readily identified as a typo were modified (i.e. 
2129659755 appears only once and no other IDs end in 755, this ID was modified to an existing 
student ID 2120659755) and included in the counts. In addition, CCR coaches included 
students who transferred in or transferred out, which in most cases aligned with the end of 
services or the initiation of services to avoid identifying those students as underserved. When 
transfer dates did not align with services provided, those students were kept in the data. In 
Table B2 below, two districts have perfect alignment between the two lists. Some discrepancy of 
alignment is expected as students move in and out of a school for a variety of reasons. It was 
established that a 10% difference would allow for these changes in services provided. With that 
level of error, nine of nine districts met fidelity for this component over the course of the project. 


For the Student Success Plans that were completed, recorded, and revised, all nine districts 
met fidelity. Plans were revised for most students more often than expected, indicating 
continued usage of the Student Success Plans to support updating of student aspiration. 


Table B1. Expectations for Strategist-Student Interactions for 2018-19 School Year 


Expectations for Student Interactions by Month 


P| Atal | Expected 
ee 
ee 
[November | 2 
ee 


ee 
ee 
ee 


May 


Total Expected Interactions a ae 


*Some districts in Kentucky ended school earlier than the third week of 
May with classes not meeting regularly during the last two weeks. 
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Table B2. Analysis of Student Count Data 


Not Aligned Total Count % Missing SSP Count % Missing 


District 1 

SSP 3 37 8% 0 0% 
Strat 8 31 2% i 
District 2 

SSP 0 29 0% 0 0% 
Strat 0 29 % i 
District 3 

SSP 1 44 2% 0 0% 
Strat 5 40 3% 
District 4 

SSP 0 57 0% 1 2% 
Strat 8 49 16% a ae 
District 5 

SSP 0 48 0% 0 0% 
Strat 0 48 % 
District 6 

SSP 0 re 0% 0 0% 
Strat 24 17 11% 
District 7 

SSP 1 18 6% 0 0% 
Strat 0 19 oo 
District 8 

SSP 0 34 0% 0 0% 
District 9 

SSP 0 40 0% 0 0% 
Strat 0 40 % i 


For the purposes of clarification, College and Career Readiness (CCR) coaches are the project 
staff assigned to each district to support project implementation, and strategists are school-level 
staff that interact with students for ten minutes each week. 


Program Supports for the CCR Coach 


In both key components associated with CCR Coach support, coaches identified training and 
collaboration as a strength of the project that allowed coaches to solve implementation issues in 
their districts as they arose. During interviews, coaches expressed support for the goals of the 
project, the training they received, and the network the coaches created. During the first two 
years of implementation, there was an expressed desire for examples of instructional strategies 
to be more varied as most were mathematics-based, and much of the work coaches did with 
students was language based. The project created a more detailed and varied curriculum that 
allowed the CCR coach to provide to strategists an outline of activities that gave students 
greater understanding of how they could achieve the project goals, but still allowed strategists to 
have flexibility to support students as needs arose. 
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All coaches received training and monthly support meetings throughout the course of the 
project. Coaches indicated these meetings provided needed support and the opportunity for 
collaboration across the coaching network to identify solutions to barriers of implementation. 


Teacher Participation 


All nine schools offered Professional Development (PD) through the project. Teachers from all 
nine schools attended the professional learning opportunities at a rate that met the fidelity 
threshold. Teachers indicated that offered programming helped them understand how to support 
high poverty and special education students more effectively during classroom instruction. 


Strategist Participation/Supports 


The project met fidelity in both indicators for Strategist supports. During year 2 interviews, 
school-level strategists offered extensive support for the training provided. As with year 1, 
feedback was mixed. More than 50% of strategists liked the addition of the binder of activities 
they could use. There was significant feedback that the binder of activities was repetitive or not 
always applicable to their students (i.e. a sophomore male from a farming community did not 
want to draw as part of his meetings). Overall, strategists felt the materials were sufficient and 
the support the CCR coaches provided was excellent. 


Table B3. Analysis of Strategist Meetings with Students Across the Four Years of Implementation 


District District District District District District District District District 


1 2 3 4 5 6 7 8 9 rete 
ee 123 99.5 138 198 155 109 57 134 163 | 1176 
TotalMigs | 3110 | 2340.5 | 3926.5 | 5355.5 | 4671.5 | 2963.5 | 1443.5 | 2954.5 | 4239.5 | 31005 
Total Ave 25.4 23.5 28.5 27.0 30.1 272 25.3 22.0 26.0 | 26.4 
Std.Met22 | 82 64 117 163 140 89 40 79 125 896 
: a 67% 64% 84% 82% 90% 82% 69% 59% 76% | 76% 


An indicator for this component is that CCR coaches will gradually release meetings with 
students to school-level strategists to promote sustainability of this project in years 3 and 4. 
Data indicates that CCR coaches in six of nine districts continued to release responsibility for 
observations. CCR Coach-led sessions declined by approximately 57% from year 2 through 
year 4. Table B4 below provides data for year 4 (2018-19 school year). 


Table B4. Analysis of Interactions by CCR Coach 


CCR Coach District CCR Total Year4,%of Year 3, % of 
Count Count Interactions Interactions 
CCR Coach 1 _ District 9 1243 103 8% 13% 
CCR Coach 2 District 8 947 156 16% 17% 
CCR Coach 2 District 7 574 58 10% 12% 
CCR Coach3 District 6 511 No CCR Coach in district 
CCR Coach4 District 5 1503 130 9% 9% 
CCR Coach 5 District 4 1770 95 5% 6% 
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CCR Coach6 District 3 1208 377 31% 6% 
CCR Coach7 District 2 830 248 30% 26% 
CCR Coach 8 District 1 692 202 29% 34% 

Total 8767 1369 15% 13% 


Community and Family Engagement 


School-based Support Team (SBST) meetings were conducted as expected and included an 
administration representative. This indicator was met with fidelity. All nine schools documented 
at least two family activities during the 2018-19 school year. In fact, family engagement activities 
expanded to average seven events per school. 


Career Exploration field trips were documented for all nine schools. During interviews with 
students and school-level strategists, they identified going on trips to visit a variety of colleges, 
including schools across the region (i.e., University of Kentucky, University of Louisville, Murray 
State University, Western Kentucky University, Southcentral Community & Technical College, 
Somerset Community College, etc.). Students spoke about visits to industry or work placement 
in support of the goals of the project (stronger connection to a career pathway). Documentation 
showed an above project average of opportunities for career explorations and college visits. In 
all, 29 College and Career Exploration trips occurred with all but one district taking more than 
two trips. 


All nine schools achieved documented levels of parent contact that met fidelity during the 2018- 


19 school year. The table below (Table B5) shows the number of events and average 
interaction rate with parents. 


Table B5. Parent/Community Events with Student/Parent Participation 


District Community % Student/Parent 
Events Participation 

District 9 a Oe 
District 8 i ae 
District 7 ‘ ae 
District 6 ae 
District 5 : ck 
District 4 a aks 
District 3 7 kG 
District 2 : kk 
District 1 5 50% 

Average 7 52% 
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Figure B1: Logic Model: 


Needs 
Assessment 
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School 
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plan 
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understanding 
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implement 
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4a. Teacher 
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positive student 
behaviors 
Coaching to remove 
barriers and reach 
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Support 
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Coach 
CCR coach gradual 
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*® Meetings with coach 
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* Weekly meetings with 
students to provide 
guidance 

* Development of an ILP 
Career Pathway Plan 


6b. SBST 
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Activities 

* Quarterly review of 
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Teachers 
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Families 
Increased family 
involvement 


Increased 
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secondary success 
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Increased access to career 
pathway experiences 
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community and to connect 
career goals and classroom 
experiences 

Increased expectations for 
own graduation 

Increased feelings of 
autonomy, influence, and 
control/increased self- 
determination and goal- 
setting behaviors 

Increased self-management 
behaviors 

Improved academic 
achievement 


Student 

* Increased CCR 
certifications/ 
benchmarks 

* Increased 
graduation rate 
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Table B6. Fidelity of Implementation Matrix for Four Years of Implementation, Scores for the 2018-19 School Year 


Key Component 1= Needs Assessment 


Data Retreat: Initial training Summer (initially 3 day, School 0 (low) = 0-1 days of attendance by | School 1 2 9 of 9 Schools participated in initial 
subsequent years 1 day) leadership team (including school __| School 2 2 data retreat with moderate and high 
leadership team, including principal and counselor) School 3 2 attendance 
school principal, and 1 = (moderate) 2 days attendance rae : : 
eouecler data analysis 2 (high) = full attendance (3 days bolic 62 
ereriell initially, 1 day subsequent years) School 7 2 

School 8 2 
School 9 2 

Initial Implementation Plan Summer leadership team data | School 0 (low) = score of 1-2 ondocument | School 1 2 9 of 9 Schools have Implementation 

analysis session review tool rubric scale School 2 2 Plans that meet the threshold of 
1 (moderate) = score of 3 School 3 2 high on Document Review Tool. 

ee School 4 2 

2 (high) = score 4-5 Scheel 52 

School 6 2 

School 7 2 

School 8 2 

School 9 2 

Follow-up Implementation Plan | 0.5 day Assessment of School 0 (low) = score of 1-2 on revised School 1 2 9 of 9 Schools have revised 
progress/ refinement, during document review tool rubric scale | School 2 2 Implementation Plans that meet the 
school year 4 (moderate) = score of 3 rene : ; threshold of moderate on the 

igh) = Document Review Tool 
2 (high) =score of 4-5 School 5 2 CSUMERE SSRN 100 
School 6 2 
School 7 2 
School 8 2 
School 9 2 
All indicators Project fidelity score range: 0-6 | School 6 2015-16 Project met fidelity 


2015-16 9 of 9 schools score 6 
2016-17 9 of 9 schools score 6 
2017-18 9 of 9 schools score 6 
2018-19 9 of 9 schools score 6 


2016-17 Project met fidelity 
2017-18 Project met fidelity 
2018-19 Project met fidelity 
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Key Component 2 = Culture Survey 


Teacher Culture Review Survey | Teacher survey Teacher 0 = Teacher does not complete | School 1 1 9 of 9 Schools meet the criteria for high 
survey School 2 1 school fidelity with at least 75% of 


1 = Teacher completes survey alee : teachers taking the survey. 


School 5 1 
School 6 1 
School 7 1 
School 8 1 


School 9 1 


Student Culture Review Survey | Student survey Student 0 = Student does not complete | School 1 1 All 9 Schools meet the criteria for high 
survey School 2 1 school fidelity with at least 75% of 


1 = Student completes survey ae : students taking the survey. 


School 5 1 
School 6 1 
School 7 1 
School 8 1 
School 9 1 


Input Record Review Record Review of IEP, ILP- | Student 0 = student record is not School 1 1 9 of 9 Schools received a score of 1. 
GRECC reviews IEPs at the reviewed in 1% review cycle School 2 1 Alignment between Strategist Records 
school level — holistically 4 = student record is reviewed in | School 3 1 and SSP records is reported in Table 2 


School 4 1 
4st | 
review cycle School 5 1 


School 6 1 
School 7 1 
School 8 1 
School 9 1 


Input Record Review 2 Lead strategist reviews ILP, | Student 0 = student record is not School 1 1 9 of 9 Schools received a score of 1 for 
CP, and Schedule alignment reviewed in 2"4 review cycle School 2 1 reviews of SSP that were initially 
4 = student record is reviewed in. | Schoo! 3 1 completed. 


) School 4 1 
2nd 
review cycle School 5 1 


School 6 1 
School 7 1 
School 8 1 
School 9 1 
Project fidelity score range: 0 - | School 4 2015-16 Project met fidelity 

4 2015-16 9 of 9 schools score 4 | 2016-17 Project did not meet fidelity 
2016-17 6 of 9 schools score 4 | 2017-18 Project met fidelity 
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2017-18 9 of 9 schools score 4 
2018-19 9 of 9 schools score 4 


2018-19 Project met fidelity 


Key Component 3a = Program Supports for the CCR Coach 


Initial Coach Training 5 day initial training of coach —_| Project 0 = project does not provide Project 1 The initial training for coaches occurred 
(year 1 only) by the program initial 5-day coach training during summer of 2015 

1 = project provides initial 5-day 
coach training 


On-going Coach Training 1 day/month training of coach | Project 0 = project does not provide Project 1 On-going training delivered effectively, 
by the program monthly ongoing coach training based on interviews with CCR coaches 
1 = project provides monthly as issues arose. Project addressed 
ongoing coach training (9 training supports as needed. 
sessions over year) 
All Year 1 Project 2 2015-16 Project met fidelity 
Project fidelity score range 0- | 2015-16 project score 2 2016-17 Project met fidelity 
2 2016-17 project score 2 2017-18 Project met fidelity 
2017-18 project score 2 2018-19 Project met fidelity 


2018-19 project score 2 


Key Component 3b = Coach Participation in PD 


Initial Coach Training 5 day initial training of coach =| Coach 0 = coach attends 1-3 days of the | Year 1, project 2 
(year 1 only) by the program 5-day initial coach training 
1 = coach attends 4 days of the 
5-day initial coach training 
2 = coach attends 5 days of the 
5-day initial coach training 


On-going Coach Training 1 day/month training of coach | Coach 0 = coach participates 0-65% of | Project 1 All 7 coaches received on-going training with 
by the program ongoing monthly training fidelity 
sessions 


1 = coach participates in >65% of 
ongoing monthly training 


sessions 

All Year 1 Project 1 2015-16 Project met fidelity 
Fidelity score range 0 -3 2015-16 project score 3 | 2016-17 Project met fidelity 
Year 2-4 2016-17 project score 2 | 2017-18 Project met fidelity 
Fidelity score range 0-1 2017-18 project score 2 | 2018-19 Project met fidelity 


2018-19 project score 2 


Key Component 4a = School/Program Supports for Teachers 
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Teacher PD School provides PD (based on | School 0: PD not delivered by school School 4 1 All 9 Schools offered PD 


needs assessment) . i School 2 1 
1: PD delivered by school School 34 


School 4 1 
School 5 1 
School 6 1 
School 7 1 
School 8 1 


School 9 1 
Coaching support for teachers | On-going coaching support for | Project 0: Less than 75% of project School 4 1 All 9 Schools offered coaching support 


teachers from CCR coach teachers indicate that they were | School 2 1 
provided by project offered coaching support School 3 1 


1: At least 75% of project one : 
teachers indicate that they were | 201100 


; School 6 1 
offered coaching support School 74 


School 8 1 
School 9 1 
All Project fidelity score range: 0 | Project 2 2015-16 Project met fidelity 
-2 2015-16 project score 2 | 2016-17 Project met fidelity 
2016-17 project score 2 | 2017-18 Project met fidelity 
2017-18 project score 2 | 2018-19 Project met fidelity 
2018-19 project score 2 


Key Component 4b = Teacher Participation 


Teacher PD Teacher uptake of PD School School fidelity 0: < 25% of School 1 1 9 of 9 Schools received a fidelity score of 1 
(Duration determined by teachers are high fidelity School 2 1 
needs assessment for each 1: 25 - 69% of teachers are high School 3 1 
school) fidelity at! : : 
2: at least 70% of teachers are sso 61 
high fidelity Paola 
School 8 1 
School 9 1 

Meetings with CCR Coach Teachers identify receiving | School School fidelity 0: < 25% of School 1 1 9 of 9 Schools scored a 1 

coaching support from CCR teachers are high fidelity School 2 1 
coaches 1: 25 - 50% of teachers are high sol : 
— School 5 
School 6 1 
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Project fidelity score range = 
0-2 
High fidelity school = 2 


2: >50% of teachers are high School 7 1 

fidelity School 8 1 

School 9 1 

All School 2 


2015-16 5 of 9 schools 
score 22 
2016-17 6 of 9 schools 
score 22 
2017-18 9 of 9 schools 
score 22 
2018-19 9 of 9 schools 
score 22 


High fidelity project = score of 


2015-16 project score 
2016-17 project score 


2017-18 project score 


Initial strategist PD Initial training for strategist Project 0: training provided for no/only Project 1 Meets fidelity score 
provided by the project some strategists 
virtually and available via 1: Training provided to all 
recording for viewing as strategists 
needed 
Strategist Coaching Ongoing project support for Project 0: strategist coaching not offered | Project 1 Meets fidelity score 
strategists as they refine 1: strategist coaching offered 
implementation of treatment 
program 
Gradual release of student CCR coaches transfer Project 0 = >30% of strategists’ student | Project 1 
meetings from coach to strategist role responsibilities load assigned to coaches 
strategist to the strategists over the last 1 = 10-30% of strategists’ student 
(Years 3 and 4 only) two years of the grant load assigned to coaches 
2 = <10% of strategists’ student 
load assigned to coaches 
All Project 3 
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Key Component 5b = 


Strategist Participation/Activities 


Strategist PD Uptake of initial PD by Strategist School fidelity 0: < 25% of School 1 2 9 of 9 Schools meet fidelity score of 2 at the 
Strategist (Duration strategists with score of 1 School 2 2 School level 
determined by needs 1: 25 — 74% of strategists with | School 32 
assessment for each school) score of 1 re : ; 
. aera 75% of strategists with rae 62 
School 7 2 
School 8 2 
School 9 2 
Strategist Coaching Strategist uptake of ongoing _| Strategist School fidelity 0: < 25% of School 1 2 9 of 9 Schools meet fidelity score of 2 at the 
project support for strategists strategists with score of 1 School 22 School level. 
as they refine implementation 1: 25-74% of strategists with | School 32 
of treatment program score of 1 School 4 2 
2: at least 75% of strategists with eee 
; School 6 2 
score of 1 School 72 
School 8 2 
School 9 2 
Weekly Meetings with students | Strategist meets with each Strategist 0=Strategist meets with < 25% of | Student Level 7 of 9 Schools meet fidelity score of 2 at the 
student for 10 minutes/week assigned students for 10 minutes | 2 = 304 students (79%) | School level. 
3 times/month or more 1 = 62 students (16%) 
1 = Strategist meets with 25-75% | 9 - 49 students (5%) 
of assigned students for 10 
minutes 3 times/month or more School Level A 
2= Strategist meets with >75% of School 1 1 (61 *) 
assigned students for10 minutes eae : : ae 
. 0 
3 times/month or more School 4 2 (91%) 
School 5 2 (92%) 
School 6 1 (39%) 
School 7 2 (94%) 
School 8 2 (85%) 
School 9 2 (93%) 
Career Pathway Plans Strategist develops a Strategist School fidelity School 12 Meets fidelity 
document with student O= <25% strategists score 1 or 2. | Schoo! 22 
career/education goals and 1=25-75% strategists score 1 or —— 
plan to achieve goals 9 School 42 
; School 5 2 
2=>75% strategists score 1 or 2 | School 6 2 
School 7 2 
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School 8 2 
School 9 2 
All Project fidelity score range= | 2015-160 of 9 schools | 2015-16 Project did not meet fidelity 
0-8 score 8 2016-17 Project did not meet fidelity 
High fidelity school = 8 2016-17 0 of 9 schools | 2017-18 Project did not meet fidelity 
across 4 indicators score 8 2018-19 Project met fidelity 
2017-18 0 of 9 schools 
score 8 
2018-19 7 of 9 schools 
score 8 
Key Component 6a= Program Support for School-based Support Team 
Support from CCR coach Program provides sufficient Project 0 = at least one project school Project 1 Meets fidelity 
trained coaches for supporting was not provided with a coach for 
all SBSTs the SBST 
1 = all project schools were 
provided with a coach for the 
SBST 
Quarterly Review Mtgs with Coach meets with SBST School 0: SBST meets 2 times a year or | School 1 2 Meets fidelity 
CCR Coach. monthly to review data and less with Coach School 2 2 
project implementation in the 4:3 meetings a year coe : ' 
i . choo 
school (approx. 30 minutes) 2: 4+ meetings a year des 
School 6 2 
School 7 2 
School 8 2 
School 9 2 
Quarterly review of project SBST meets monthly and School 0: SBST holds 1-2 meetings per _| School 1 2 Meets fidelity 
implementation and data reviews implementation year to review implementation — | School 22 
concerns and data 1: 3 meetings per year School 3 2 
Ate neal School 4 2 
: 4+ meetings per year Shoals o 
School 6 2 
School 7 2 
School 8 2 
School 9 2 
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High fidelity school = score of 
2, with each indicator with a 
score of > 1 


2015-16 9 of 9 schools 
score 22 
2016-17 9 of 9 schools 
score 22 
2017-18 9 of 9 schools 
score 22 
2018-19 9 of 9 schools 
score 22 


2015-16 Project met fidelity 
2016-17 Project met fidelity 
2017-18 Project met fidelity 
2018-19 Project met fidelity 


Key Component 7a = Community and Family Engagement Opportunities Provided by Schools 


trips 


Parent Engagement Activities | School implements 2 family —_| School 0 = 0-1 parent engagement School 1 1 Meets fidelity 
engagement activities per year activity per year School 2 1 
1 = 2 or more parent — : 
it activiti 
engagement activities per year | o 
School 6 1 
School 7 4 
School 8 1 
School 9 4 
SBST Meetings with SBST participates in 2 School 0= SBST participated in no School 1 1 Meets fidelity 
Families/Community meetings per year with family meetings with parents/year School 2 1 
and community 4 = SBST participated in 4 eae ; 
meeting with parents een 5 1 
2= SBST participated in 2+ School 6 1 
meetings with parents School 7 4 
School 8 4 
School 9 1 
Family Contact School staff has contact with | School 0:0-33% parents report > 4 School 12 9 of 9 Schools meet fidelity score of at least a 1 
family through intentional contacts from school School 2 2 
outreach 1: 34-66% parents report >4 rile : : 
contacts from school ese 5S 
2: >66% of parents report >4 School 6 2 
contacts from school School 72 
School 8 2 
School 9 4 
Field Trips School organizes field trips for | School 0 = school organizes 0-1 field School 1 0 7 of 9 meet fidelity with at least a 1. 
students to tech ctrs and trips School 2 2 
colleges 1 = school izes 2-3 field School 3 2 
school organizes 2-3 fie ania 
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2 = school organizes 4 or more 
field trips 


School 5 2 
School 6 2 
School 7 2 
School 8 1 
School 9 1 


Key Component 7b. Community and Family Participatio 


school = score 3 or higher, 
with score of at least 1 on all 
indicators 


2015-16 7 of 9 schools 
score 23 with at least 
a1 in each indicator 
2016-17 8 of 9 schools 
score 23 with at least 
a in each indicator 
2017-18 7 of 9 schools 
score 23 with at least 
a1 in each indicator 
2018-19 7 of 9 schools 
score 23 with at least 
a1 in each indicator 


2015-16 Project met fidelity 
2016-17 Project met fidelity 
2017-18 Project met fidelity 
2018-19 Project met fidelity 


Attendance at involvement Family attends events planned | Family School fidelity School 12 9 of 9 Schools meet fidelity score of at least a 1 
activities by school to support GTP 0= 0-25% families score 1 School 2 2 
initiatives 1= 26-50% families score 1 baad : : 
2= >50% families score 1 School 5 2 
School 6 2 
School 7 2 
School 8 2 
School 9 2 
Field Trips School community supports | School 0 = school has no community School 1 1 Meets fidelity 
school career pathways field connections to any field trips School 2 1 
trips 1 = school has community School 3 1 
connections to at least one erat! : 
school field trip eciasle4 
School 7 1 
School 8 1 
School 9 1 


High fidelity school = score 
>2, at least a 1 for each 
indicator 


2015-16 2 of 9 schools 
score =2 with at least 
a in each indicator 


2015-16 Project did not meet fidelity 
2016-17 Project did not meet fidelity 
2017-18 Project met fidelity 
2018-19 Project met fidelity 
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